Estimating Caloric Expenditure using Heart Rate Model Specific to Motion Class

ABSTRACT

Embodiments are disclosed for estimating caloric expenditure using a heart rate model specific to a motion class. In an embodiment, a method comprises: obtaining acceleration and rotation rate from motion sensors of a wearable device; determining a vertical component of inertial acceleration and a vertical component of rotational acceleration from the acceleration and rotation rate, respectively; determining a magnitude of the rotation rate; determining a correlation between the inertial vertical acceleration component and rotational acceleration; determining a percentage of motion outside a dominant plane of motion; predicting a motion class based on a motion classification model that takes as input the motions, correlation and percentage; determining a likelihood the user is walking; in accordance with determining that the user is likely not walking, configuring a heart rate model based on the predicted motion class; and estimating, using the configured heart rate model, a caloric expenditure of the user.

TECHNICAL FIELD

This disclosure relates generally to activity monitoring using wearabledevices.

BACKGROUND

The metabolic equivalent of task (MET) is defined as a ratio of the rateof energy expended by an individual during physical activity to the rateof energy expended by the user at rest (referred to as the restingmetabolic rate (RMR)). Many studies have shown that the conventional1-MET baseline overestimates actual resting oxygen consumption andenergy expenditures by about 20% to 30% on average. Therefore, anaccurate calculation of MET for a specific individual requires dataspecific to the user and the activity.

Modern wearable devices (e.g., smart watches, fitness bands) are oftenused by individuals during fitness activities to determine their caloricexpenditure during the fitness activity. Some wearable devices includeinertial sensors (e.g., accelerometers, angular rate sensors) that areused to estimate a work rate (WR) based MET for the user wearing thedevice. Some wearable devices include a heart rate (HR) sensor thatprovides HR data that can be used with user estimated maximal oxygenconsumption (VO₂Max) and other data (e.g., users weight, age) toestimate HR based MET.

Because studies have shown that upper limb exercise leads to higherheart rate than lower limb motion there is a need for different HRmodels for upper limb exercise and lower limb exercise and thus a needto detect upper and lower limb motion.

SUMMARY

Embodiments are disclosed for estimating caloric expenditure using aheart rate model specific to a motion class.

In an embodiment, a method comprises: obtaining, using one or moreprocessors, acceleration and rotation rate from motion sensors of awearable device worn on a limb of a user while the user is engaged in aphysical activity; determining, using the one or more processors, avertical component of inertial acceleration and a vertical component ofrotational acceleration from the acceleration and rotation rate,respectively; determining, using the one or more processors, a magnitudeof the rotation rate; determining, using the one or more processors, acorrelation between the vertical component of inertial acceleration andthe vertical component of rotational acceleration rate; determining,using the one or more processors, a percentage of motion outside adominant plane of motion; predicting, using the one or more processors,a motion class based on a motion classification model that takes asinput the vertical components of inertial acceleration and rotationalacceleration, the magnitude of rotation rate, the correlation betweenthe vertical component of the inertial acceleration and the verticalcomponent of rotational acceleration and the percentage of motionoutside the dominant motion plane; determining, using the one or moreprocessors, a likelihood the user is walking; in accordance withdetermining that the user is likely not walking, configuring, using theone or more processors, a heart rate model based on the predicted motionclass; and estimating, using the configured heart rate model, a caloricexpenditure of the user.

In an embodiment, determining, using the one or more processors, alikelihood that the user is walking, further comprises: obtaining, froma digital pedometer, a step count; determining an arm pose of the user;and determining whether or not the user is walking based on the stepcount and arm pose.

In an embodiment, the motion classification model outputs one of threepossible classes: arm only motion, with body motion and other motion.

In an embodiment, the motion classification model has two parts: a firstpart that uses a logistic regression model with vertical acceleration,vertical component of the rotational acceleration, and correlationbetween the two acceleration to predict a likelihood of arm motion only,and a second part that detects a body component in the motion with amoderate likelihood from the logistic regression model, wherein if thepercentage of motion outside of the dominant plane is above a thresholdand the inertial vertical acceleration is within an expected range ofbody motion, the classification is with body motion.

In an embodiment, configuring, using the one or more processors, a heartrate model based on the predicted motion class, further comprises:determining a first caloric expenditure based on a heart rate model;obtaining a scale factor based on the predicted motion class; andscaling the first caloric expenditure by the scale factor to get asecond caloric expenditure specific to the motion class.

In an embodiment, the predicted motion class is one of a plurality ofmotion classes including arm only motion, with body motion and othermotion.

In an embodiment, determining a first caloric expenditure based on aheart rate model, further comprises: obtaining the user's age; obtainingthe user's heart rate from a heart rate sensor embedded in or attachedto the wearable device; obtaining the user's maximal oxygen uptake(VO₂Max); and determining the first caloric expenditure based on theheart rate model with the user's age, the users heart rate and theuser's VO₂Max as inputs into the heart rate model. In the case where theuser's age and VO₂Max are not available a default caloric expenditure isused.

In an embodiment, the rotation rate is compensated for drift.

In an embodiment, the dominant plane of motion is determined usingprinciple component analysis (PCA) of a crown vector of the wearabledevice, where the dominant plane is determined by first and second PCAcomponents and the percentage of motion outside of the dominant plane isa fraction of a variance in a third PCA component, which is a ratio ofthe variance in the third component and a total variance.

In an embodiment, a system comprises: one or more processors; memorystoring instructions that when executed by the one or more processors,cause the one or more processors to perform operations comprising:obtaining acceleration and rotation rate from motion sensors of awearable device worn on a limb of a user while the user is engaged in aphysical activity; determining a vertical component of inertialacceleration and a vertical component of rotational acceleration fromthe acceleration and rotation rate, respectively; determining amagnitude of the rotation rate; determining a correlation between theinertial vertical acceleration component and rotational acceleration;determining a percentage of motion outside a dominant plane of motion;predicting a motion class based on a motion classification model thattakes as input the vertical components of inertial acceleration androtational acceleration, the magnitude of rotation rate, the correlationbetween the vertical components of the inertial acceleration androtational acceleration and the percentage of motion outside thedominant motion plane; determining a likelihood that the user iswalking; in accordance with determining that the user is likely notwalking, configuring a heart rate model based on the predicted motionclass; and estimating, using the configured heart rate model, a caloricexpenditure of the user.

Other embodiments can include an apparatus, computing device andnon-transitory, computer-readable storage medium.

The details of one or more implementations of the subject matter are setforth in the accompanying drawings and the description below. Otherfeatures, aspects and advantages of the subject matter will becomeapparent from the description, the drawings and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a compound exercise where the participant is liftingdumbbells while performing side squats, according to an embodiment.

FIG. 1B illustrates three-dimensional (3D) motion in a wearable devicebody frame for side squat plus arm raise and front arm raise, accordingto an embodiment.

FIG. 2A is a scatter plot illustrating normalized METs versus 1-fHR fora body workout, according to an embodiment.

FIG. 2B is a scatter plot illustrating normalized METs versus 1-fHR foran arm only workout, according to an embodiment.

FIG. 3 is a block diagram of a system for determining caloricexpenditure using a HR rate model specific to motion class, according toan embodiment.

FIG. 4 illustrates a motion classification model, according to anembodiment.

FIG. 5A is a scatter plot illustrating normalized METs versus 1-fHR fora body workout, according to an embodiment.

FIG. 5B is a scatter plot illustrating normalized METs versus 1-fHR foran arm only workout, according to an embodiment.

FIG. 5C is a scatter plot illustrating normalized METs versus 1-fHR forother types of workouts, according to an embodiment.

FIG. 6 is a flow diagram of a process of determining caloric expenditureusing a HR rate model specific to a motion class, according to anembodiment.

FIG. 7 is example wearable device architecture for a wearable deviceimplementing the features and operations described in reference to FIGS.1-6.

DETAILED DESCRIPTION Problem Statement

FIG. 1A illustrates a compound exercise where the participant is liftingdumbbells while performing side squats, according to an embodiment. Inthis example, wearable device 101 is worn on the wrist of participant100. Wearable device 101 can be, for example, a smart watch or fitnessband or any other device that can measure accelerations and rotationrate. In an embodiment, wearable device 101 can have an architecture asshown in FIG. 7, which includes a 3-axis MEMs accelerometer formeasuring acceleration and a 3-axis MEMS gyro for measuring angularrates. During the exercise, participant 100 is lifting dumbbells andsquatting from side-to-side. The lifting of the dumbbells is an exampleof upper limb motion and the side squats are an example of lower limbmotion. Using an HR model designed for whole body workouts (like walkingand running) would overestimate the calorie expenditure of the user whenthe user engages only their upper limb in the exercise and their HRincreases. Accordingly, it is desirable to use different HR models forupper limb motion and lower limb motion, which requires that the upperlimb motion be distinguished from the lower limb motion so that theappropriate HR model is used to determine the user's total calorieexpenditure for the exercise.

As can be observed from FIG. 1B, when using solely the upper body toperform front raises, there is a single dominant plane of motion. Whenperforming the compound motion of side squats and front raises usingboth the upper and lower body simultaneously, the wearable deviceorientation takes a more complex trajectory and there is not one cleardominant plane of motion. Note that the signal/peaks appearing on theright outside of the boxed region are from side raises

FIG. 2A is a scatter plot illustrating normalized METs versus 1-fHR fora body only workout, according to an embodiment. For this plot 50 datapoints were plotted for a 0.5 min to 1 min non-anaerobic exercisesegment with stable METs (Note: NormalizedMETS=METs/VO₂Max). Linearregression analysis shows that HR model 201 (represented by the straightline) used to compute the METs is valid for body only workouts becausemost of the data points fall within the +/−20% linearity uncertaintybound represented by the dashed lines. Note thatfHR=(hr_max−hr)/(hr_max−hr_min), where hr=heart rate measurement,hr_max=user's estimated maximum heart rate hr_min=user's estimatedminimum heart rate, thus if 1-fHR=0.3, then that is 30% of the way tothe maximum heart rate.

FIG. 2B is a scatter plot illustrating normalized METs versus 1-fHR foran arm only workout, according to an embodiment. For this plot 50 datapoints were plotted for a 0.5 min to 1 min non-anaerobic exercisesegment with stable METs. Linear regression analysis shows that HR model201 used to compute the METs is not valid for arm only workouts becausemost of the data points fall outside the +/−20% linearity uncertaintybound represented by the dashed lines.

Accordingly, FIGS. 2A and 2B illustrate that at least two different HRmodels are needed to accurately calculate HR based METs for compoundworkouts: one for workouts with body motion and one for workouts witharm only motion. Some examples of arm only workouts include but are notlimited to: ball down curl press, bicep curl, bicep curl with rotation,front raise, front and lateral raise, lat pulldown, lateral raise, lowchest fly, rear delt pull, upper body row and up right row. Someexamples of body motion workouts include but are not limited to: latpull down with squat, lunges, lunges plus row, side squat lunges, sidesquat plus raise, squat, squat with ball and squat with lunges. Examplesof exercises that are not arm only motion or body motion include but arenot limited to: plank walkout, pushup with side plank and Russiantwists.

System Overview

FIG. 3 illustrates a system 300 for determining caloric expenditureusing a HR rate model specific to a motion class, according to anembodiment. System 300 can be implemented in a wearable device, such asa smartwatch or fitness band worn on a wrist of a user during a physicalactivity, such as a workout. Although the embodiments that followdescribe exercises, the disclosed embodiments are applicable to anyphysical activity that where an caloric expenditure is desired.

In an embodiment, the wearable device includes motion sensors, such asaccelerometer 301 (e.g., a 3-axis MEMS accelerometer) and angular ratesensor 302 (e.g., 3-axis MEMS gyro) that provide a three-dimensional(3D) acceleration vector and 3D rotation rate vector in a body frame ofthe wearable device, respectively. The 3D acceleration and rotation ratevectors are input into motion processor 303, which computes and outputsa 3D acceleration vector with gravity removed, a gravity vector and a 3Drotation rate vector that is compensated for drift.

The output of motion processor 303 is input into physics/statisticsprocessor 304, which computes and outputs a vertical component ofinertial acceleration in an inertial world coordinate frame(hereinafter, referred to as “inertial frame”), a rotation ratemagnitude, a vertical component of rotational acceleration in theinertial coordinate frame due to the rotation of the user's limb and acorrelation coefficient representing the correlation between thevertical component of translational acceleration and the verticalcomponent of rotational acceleration.

In an embodiment, physics/statistics processor 304 uses a coordinatetransformation (e.g., a direction cosine matrix or quaternion) that usesyaw, pitch and roll angles derived from the 3D rotation rate totransform the 3D acceleration vector (with gravity removed) from thewearable device body frame to the inertial frame, where the verticalcomponent of inertial acceleration is normal to the Earth surface. Therotation magnitude is computed by taking the Euclidean norm of therotation rate vector. In an embodiment, the correlation coefficient isdirectly calculated using the Pearson correlation equation, betweeninertial vertical acceleration and the inertial vertical rotationalacceleration. Also computed by physics/statistics processor 304 is apercentage of motion outside the dominant plane and the user's limb pose(e.g., arm pose).

In an embodiment, the dominant plane of motion is determined usingprinciple component analysis (PCA) of a crown vector of the wearabledevice per measurement epoch. The first and second principal componentscharacterize the dominant plane. The third principal componentrepresents motion outside the dominant plane. The eigenvalues of eachprincipal component represent the magnitude of motion in the directionof its respective component. The fraction of motion outside the dominantplane is quantified by a ratio of the third eigenvalue to the L1 norm(sum of absolute values) of all three eigenvalues. In an embodiment, theuser's arm pose is defined by the angle between the crown vector and thehorizontal direction (the direction perpendicular to gravity).

The vertical component of inertial acceleration, rotation ratemagnitude, the vertical component of rotational acceleration, thecorrelation between the vertical component of inertial acceleration andthe vertical component of rotational acceleration, the percentage ofmotion outside the dominant plane and the arm pose are inputs intomotion classification model 305, which predicts a motion class based onthe input. In an embodiment, motion classification model 305 predictsone of three motion classes: with body motion, arm only motion andother, as described in further detail in reference to FIG. 4.

In an embodiment, motion classification model 305 has two parts: alogistic regression with vertical acceleration, vertical component ofthe rotational acceleration, and correlation between the twoaforementioned accelerations. The output from the logistic regressionmodel is the likelihood to be without body motion (arm motion only)workout, as illustrated by FIG. 4. The second part of motionclassification model 305 is to further detect the body component in themotion with a moderate likelihood from the logistic regression model. Ifthe percentage of motion outside of the dominant plane is above athreshold and the inertial vertical acceleration is in expected range ofbody motion, the classification would be with body.

Other embodiments can include more or fewer motion classes. Any suitablemotion classification model can be used to predict motion classes,including machine learning algorithms.

FIG. 4 shows a histogram with probabilities and corresponding cumulativedistribution functions for body motion and arm only motion, according toan embodiment. The probabilities and CDFs are represented by the y-axisand likelihoods to be without (w/o) body motion are represented by thex-axis. CDF 401 is associated with the w/body motion portion of thehistogram and CDF 402 is associated with the arm only motion portion ofthe histogram. The probabilities of the predicted motion classes wereestimated from 230 collections of exercises with body motion and armonly motion. As shown in FIG. 4, if there is a likelihood of 0.3 orlower the detected motion is w/o body motion, the motion isclassified/labeled as “w/body motion.” If the likelihood is 0.8 orabove, the motion is classified/labeled as “arm only motion.” Otherwise,the motion is classified/labeled as “other” class. The three classesshown are based on a true label. The histogram of FIG. 4 shows that thelikelihood from the logistic regression model provides a good separationbetween the classes, especially between the w/o body motion and thew/body motion classes. Thresholds are set based on this separation. Forexample, as shown in the plot, if the likelihood is below 0.3, themotion is likely to be w/body motion, and less likely to be w/o bodymotion.

FIGS. 5A-5C are scatter plots showing linear regression analysis forbody workouts, arm only workouts and other workouts, respectively. Eachplot includes 50 data points of HR and normalized METS collected viareference devices. The solid lines represent the fitted HR models 502and 503 are the w/body HR model scaled by a fraction of the engagedmuscle size to the big muscle in the leg. In an embodiment, the HR modelcalculates HR based METS for each class is calculated using Equation[1]:

METS_(exercise) =R _(exercise) *f(fHR_(exercize))*VO₂MAX,  [1]

where f(fHR_exercise)) is any known HR model that works well for withtotal body exercise (e.g., running, jogging) and R_(exercize) representsa fraction of the engaged muscle size to the larger muscles in the leg,and is in a range of 0.0 to 1.0, the actual values of which can bedetermined empirically. By applying Equation [1], the HR based METvalues output by the HR model for total body exercise are reduced, thuscorrecting the overestimate of HR based MET values when the exerciseengages only the upper limb or certain exercises that engage abdominalsor back muscles.

Referring back to FIG. 3, walking detection processor 307 receives astep count and limb pose. In an embodiment, the step count is providedby digital pedometer 306. Digital pedometer 306 counts steps based onthe 3D acceleration vector in the wearable device body frame. Forexample, digital pedometer 306 can count steps by, for example, countingpeaks and/or valleys in a plot of acceleration magnitudes, usingfrequency spectrum analysis, using machine learning, or any combinationof the foregoing or any other suitable digital pedometer algorithm. Thestep count and limb pose (e.g., the angle of the arm with respect togravity) are input into walking detector 307, which determines if theuser is walking. If walking detector processor 307 detects steps, and atthe same time the user's arm is pointing down most of the time, then themotion is detected as w/ body and the w/ body heart rate model is used.This feature covers the potential cases of the user walking aroundduring exercise.

The motion class and a walking detection signal are input into HR model308, which is configured according to the motion class. Also input intothe HR model 308 is HR data from HR sensor 309, the user's age and theuser's estimated VO₂Max. In an embodiment, HR sensor 309 measures theuser's HR, which is input into HR model 308. In an embodiment, HR sensor309 is embedded in a wearable device and comprises a number of lightemitting diodes (LEDs) paired with photodiodes that can sense light. TheLEDs emit light toward a user's body part (e.g., the user's wrist), andthe photodiodes measure the reflected light. The difference between thesourced and reflected light is the amount of light absorbed by theuser's body. Accordingly, the user's heart beat modulates the reflectedlight, which can be processed to determine the user's HR. The measuredHR can be averaged over time to determine an average HR, which inputinto HR model 308.

In an embodiment, HR model 308 estimates caloric expenditure (e.g.,METs) during steady-state aerobic exercise using a relationship betweenheart rate and the user's estimated VO₂MAX. As previously discussed, HRModel 308 includes three models for different motion classes, asdescribed in reference to FIG. 5. During steady-state aerobic exercise,oxygen is utilized at a relatively consistent rate depending on theintensity of the exercise. There is an observable and reproduciblerelationship between heart rate and the calories burned from consumedVO₂. When workload intensity increases, HR increases and vice versa. Ifthe user's resting HR, maximum HR, user's VO₂ MAX and age (andoptionally weight for some models) are known, caloric expenditure (METs)can be estimated based on a percentage of the user's maximum HR or apercentage of the user's HR reserve.

In an embodiment, HR model 308 takes as input calibrated VO₂MAX and theuser's HR. The calibrated VO₂ MAX can be computed using work rate (WR)MET values (which are implicit estimates of VO₂) with the user'smeasured HR data. The output of HR model 308 are HR based MET values.The HR based MET values can be used for any desired purpose, includingbut not limited to: presentation on a display of the wearable device bya fitness or other application as an estimate of caloric expenditure,sent to another device for display and/or further processing by one ormore other applications, stored on the wearable device for subsequentdisplay/processing by one or more applications, or sent to anetwork-based computer system (the “cloud”), where the HR based METvalues can be displayed/processed by other devices connected to thenetwork-based computer system.

In an embodiment, HR based MET values are estimated over successive,non-overlapping intervals of time referred to herein as “epochs.” An“epoch” can be x seconds (e.g., 2.56 seconds, corresponding to 256samples of sensor data sampled at 100 Hz). In an embodiment, the HRbased MET can be combined (e.g., averaged) with work rate (WR) based METvalues.

Example Process

FIG. 6 is a flow diagram of process 600 of determining caloricexpenditure using a HR rate model specific to a motion class, accordingto an embodiment. Process 600 can be implemented using the wearabledevice architecture 700 disclosed in reference to FIG. 7.

Process 600 includes obtaining acceleration and rotation rate from awearable device worn on a limb of a user while the user is engaged in aphysical activity (601). For example, the wearable device can be a smartwatch or fitness band worn on the wrist of the user during a physicalactivity. Acceleration can be sensed by a 3-axis MEMS accelerometer thatoutputs a 3D acceleration vector. Rotation rate can be sensed by a3-axis MEMS gyroscope that outputs a 3D rotation rate vector, which iscompensated for drift. In an embodiment, a motion processor computes the3D acceleration vector with gravity removed, the gravity vector and 3Drotation rate in body frame of the wearable device that is compensatedfor drift.

Process 600 continues by determining vertical components (Az) ofinertial acceleration and rotational acceleration from the 3Dacceleration and compensated rotation rate (602). For example, the rawacceleration and rotation rate output by the sensors 301, 302 can betransformed into an inertial world frame, and the vertical component isthe motion that moves in a vertical plane normal to the ground. In anembodiment, the rotational acceleration is estimated from the rotationrate and a crown vector (directional vector from elbow to hand in aninertial reference frame) provided by motion processor 303.

Process 600 continues by determining a magnitude of the rotation rate(603). The magnitude can be computed by taking the absolute value of therotation rate vector output by gyro sensor 302.

Process 600 continues by determining a correlation between the verticalcomponents of the translational and rotational acceleration (604). Forexample, in additional to translation acceleration there is anadditional component of vertical acceleration due to rotation of theuser's limb. In an embodiment, the correlation coefficient is calculateddirectly using a Pearson correlation coefficient.

Process 600 continues by determining a percentage of motion outside adominant plane of motion from the crown vector, which is the vector ofcrown direction in an inertial reference frame (605). For example, PCAcan be applied to the crown vector, which gives the three independentcomponents in order based on the variance in each. The percentage out ofthe dominant plane is the ratio of the variance of the third componentto the total variance. The third component is perpendicular to thedominant plane determined by the first two components.

Process 600 continues by predicting, using a motion classificationmodel, a motion class label based on the vertical components of inertialacceleration and rotational acceleration, the magnitude of rotationrate, the correlation between the vertical components of thetranslational and rotational acceleration and the percentage of motionoutside the dominant motion plane (606). For example the motionclassification model can be a logistic regression model that predictsone of three possible motion classes: “w/body” motion class, an “armonly” motion class and “other” motion that includes all other types ofdetected motion, as described in reference to FIG. 4.

Process 600 continues by determining whether or not the user is walking(607). For the user's step count and arm pose can be used to determinewhether or not the user is walking. If the user is walking then ageneric HR model is used to calculate caloric expenditure based onuser's HR, age and estimated VO₂Max. If walking is not detected, then aHR model specific to the motion class predicted by motion classificationmodel 305 is used to calculate caloric expenditure.

Process 600 continues by configuring a HR model based on the motionclass label and whether (608). For example, while the user is engaged ina physical activity multiple acceleration and rotation rate measurementsare taken and the motion is predicted based on the vertical translationand rotational components of acceleration and a correlation between thevertical components. The predicted motion class is then used to selector determine a scale factor that can be applied to caloric generated bya generic HR model, for example, that estimates HR based METs based onthe user's HR, the user's age and the user's estimated VO₂Max.

Process 600 continues by estimating, using the configured heart ratemodel, a caloric expenditure of the user (609). For example, the scalefactor can be applied to the MET values output by the generic HR model.In an embodiment, caloric expenditure is estimated when the user may bewalking and when the user may be standing in place. The walkingdetection allows “with body” motion to be recovered that may have beenmissed. For example, if the user is likely to be walking, the “withbody” HR model is used. Otherwise, the HR model for the previous motionclassification is used, which could be any of arm only, with body,other.

Once the HR based caloric expenditure is calculated it can be used byother applications (e.g., fitness applications) running on the wearabledevice or another device, displayed on the wearable device or anotherdevice, sent to another device through a local area network, ad hocwireless network or wide area network (e.g., the Internet) to otherdevices. The HR based METS can be combined (e.g., averaged) with workrate based METs that are computed based on a work rate model thataccounts for the amount of energy expended by the user due to thephysical activity.

Exemplary Wearable Computer Architecture

FIG. 7 illustrates example wearable device architecture 700 implementingthe features and operations described in reference to FIGS. 1-6.Architecture 700 can include memory interface 702, one or more hardwaredata processors, image processors and/or processors 704 and peripheralsinterface 706. Memory interface 702, one or more processors 704 and/orperipherals interface 706 can be separate components or can beintegrated in one or more integrated circuits.

Sensors, devices and subsystems can be coupled to peripherals interface706 to provide multiple functionalities. For example, one or more motionsensors 710, light sensor 712 and proximity sensor 714 can be coupled toperipherals interface 706 to facilitate motion sensing (e.g.,acceleration, rotation rates), lighting and proximity functions of thewearable device. Location processor 715 can be connected to peripheralsinterface 706 to provide geo-positioning. In some implementations,location processor 715 can be a GNSS receiver, such as the GlobalPositioning System (GPS) receiver. Electronic magnetometer 716 (e.g., anintegrated circuit chip) can also be connected to peripherals interface706 to provide data that can be used to determine the direction ofmagnetic North. Electronic magnetometer 716 can provide data to anelectronic compass application. Motion sensor(s) 710 can include one ormore accelerometers and/or gyros configured to determine change of speedand direction of movement of the wearable device. Barometer 717 can beconfigured to measure atmospheric pressure around the mobile device.

Heart rate monitoring subsystem 720 for measuring the heartbeat of auser who is wearing the device on their wrist. In an embodiment,subsystem 720 includes LEDs paired with photodiodes for measuring theamount of light reflected from the wrist (not absorbed by the wrist) todetect a heartbeat.

Communication functions can be facilitated through wirelesscommunication subsystems 724, which can include radio frequency (RF)receivers and transmitters (or transceivers) and/or optical (e.g.,infrared) receivers and transmitters. The specific design andimplementation of the communication subsystem 724 can depend on thecommunication network(s) over which a mobile device is intended tooperate. For example, architecture 700 can include communicationsubsystems 724 designed to operate over a GSM network, a GPRS network,an EDGE network, a Wi-Fi™ network and a Bluetooth™ network. Inparticular, the wireless communication subsystems 724 can includehosting protocols, such that the mobile device can be configured as abase station for other wireless devices.

Audio subsystem 726 can be coupled to a speaker 728 and a microphone 730to facilitate voice-enabled functions, such as voice recognition, voicereplication, digital recording and telephony functions. Audio subsystem726 can be configured to receive voice commands from the user.

I/O subsystem 740 can include touch surface controller 742 and/or otherinput controller(s) 744. Touch surface controller 742 can be coupled toa touch surface 746. Touch surface 746 and touch surface controller 742can, for example, detect contact and movement or break thereof using anyof a plurality of touch sensitivity technologies, including but notlimited to capacitive, resistive, infrared and surface acoustic wavetechnologies, as well as other proximity sensor arrays or other elementsfor determining one or more points of contact with touch surface 746.Touch surface 746 can include, for example, a touch screen or thedigital crown of a smart watch. I/O subsystem 740 can include a hapticengine or device for providing haptic feedback (e.g., vibration) inresponse to commands from processor 704. In an embodiment, touch surface746 can be a pressure-sensitive surface.

Other input controller(s) 744 can be coupled to other input/controldevices 748, such as one or more buttons, rocker switches, thumb-wheel,infrared port and USB port. The one or more buttons (not shown) caninclude an up/down button for volume control of speaker 728 and/ormicrophone 730. Touch surface 746 or other controllers 744 (e.g., abutton) can include, or be coupled to, fingerprint identificationcircuitry for use with a fingerprint authentication application toauthenticate a user based on their fingerprint(s).

In one implementation, a pressing of the button for a first duration maydisengage a lock of the touch surface 746; and a pressing of the buttonfor a second duration that is longer than the first duration may turnpower to the mobile device on or off. The user may be able to customizea functionality of one or more of the buttons. The touch surface 746can, for example, also be used to implement virtual or soft buttons.

In some implementations, the mobile device can present recorded audioand/or video files, such as MP3, AAC and MPEG files. In someimplementations, the mobile device can include the functionality of anMP3 player. Other input/output and control devices can also be used.

Memory interface 702 can be coupled to memory 750. Memory 750 caninclude high-speed random access memory and/or non-volatile memory, suchas one or more magnetic disk storage devices, one or more opticalstorage devices and/or flash memory (e.g., NAND, NOR). Memory 750 canstore operating system 752, such as the iOS operating system developedby Apple Inc. of Cupertino, Calif. Operating system 752 may includeinstructions for handling basic system services and for performinghardware dependent tasks. In some implementations, operating system 752can include a kernel (e.g., UNIX kernel).

Memory 750 may also store communication instructions 754 to facilitatecommunicating with one or more additional devices, one or more computersand/or one or more servers, such as, for example, instructions forimplementing a software stack for wired or wireless communications withother devices. Memory 750 may include graphical user interfaceinstructions 756 to facilitate graphic user interface processing; sensorprocessing instructions 758 to facilitate sensor-related processing andfunctions; phone instructions 760 to facilitate phone-related processesand functions; electronic messaging instructions 762 to facilitateelectronic-messaging related processes and functions; web browsinginstructions 764 to facilitate web browsing-related processes andfunctions; media processing instructions 766 to facilitate mediaprocessing-related processes and functions; GNSS/Location instructions768 to facilitate generic GNSS and location-related processes andinstructions; and heart rate instructions 770 to facilitate hear ratemeasurements. Memory 750 further includes activity application (e.g., afitness application) instructions for performing the features andprocesses described in reference to FIGS. 1-6.

Each of the above identified instructions and applications cancorrespond to a set of instructions for performing one or more functionsdescribed above. These instructions need not be implemented as separatesoftware programs, procedures, or modules. Memory 750 can includeadditional instructions or fewer instructions. Furthermore, variousfunctions of the mobile device may be implemented in hardware and/or insoftware, including in one or more signal processing and/or applicationspecific integrated circuits.

The described features can be implemented advantageously in one or morecomputer programs that are executable on a programmable system includingat least one programmable processor coupled to receive data andinstructions from, and to transmit data and instructions to, a datastorage system, at least one input device, and at least one outputdevice. A computer program is a set of instructions that can be used,directly or indirectly, in a computer to perform a certain activity orbring about a certain result. A computer program can be written in anyform of programming language (e.g., SWIFT, Objective-C, C#, Java),including compiled or interpreted languages, and it can be deployed inany form, including as a stand-alone program or as a module, component,subroutine, a browser-based web application, or other unit suitable foruse in a computing environment.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular embodiments of particular inventions.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable sub combination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to a subcombination or variation of a sub combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

As described above, some aspects of the subject matter of thisspecification include gathering and use of data available from varioussources to improve services a mobile device can provide to a user. Thepresent disclosure contemplates that in some instances, this gathereddata may identify a particular location or an address based on deviceusage. Such personal information data can include location-based data,addresses, subscriber account identifiers, or other identifyinginformation.

The present disclosure further contemplates that the entitiesresponsible for the collection, analysis, disclosure, transfer, storage,or other use of such personal information data will comply withwell-established privacy policies and/or privacy practices. Inparticular, such entities should implement and consistently use privacypolicies and practices that are generally recognized as meeting orexceeding industry or governmental requirements for maintaining personalinformation data private and secure. For example, personal informationfrom users should be collected for legitimate and reasonable uses of theentity and not shared or sold outside of those legitimate uses. Further,such collection should occur only after receiving the informed consentof the users. Additionally, such entities would take any needed stepsfor safeguarding and securing access to such personal information dataand ensuring that others with access to the personal information dataadhere to their privacy policies and procedures. Further, such entitiescan subject themselves to evaluation by third parties to certify theiradherence to widely accepted privacy policies and practices.

In the case of advertisement delivery services, the present disclosurealso contemplates embodiments in which users selectively block the useof, or access to, personal information data. That is, the presentdisclosure contemplates that hardware and/or software elements can beprovided to prevent or block access to such personal information data.For example, in the case of advertisement delivery services, the presenttechnology can be configured to allow users to select to “opt in” or“opt out” of participation in the collection of personal informationdata during registration for services.

Therefore, although the present disclosure broadly covers use ofpersonal information data to implement one or more various disclosedembodiments, the present disclosure also contemplates that the variousembodiments can also be implemented without the need for accessing suchpersonal information data. That is, the various embodiments of thepresent technology are not rendered inoperable due to the lack of all ora portion of such personal information data. For example, content can beselected and delivered to users by inferring preferences based onnon-personal information data or a bare minimum amount of personalinformation, such as the content being requested by the deviceassociated with a user, other non-personal information available to thecontent delivery services, or publicly available information.

What is claimed is:
 1. A method comprising: obtaining, using one or moreprocessors, acceleration and rotation rate from motion sensors of awearable device worn on a limb of a user while the user is engaged in aphysical activity; determining, using the one or more processors, avertical component of inertial acceleration and a vertical component ofrotational acceleration from the acceleration and rotation rate,respectively; determining, using the one or more processors, a magnitudeof the rotation rate; determining, using the one or more processors, acorrelation between the vertical component of inertial acceleration andthe vertical component of rotational acceleration; determining, usingthe one or more processors, a percentage of motion outside a dominantplane of motion; predicting, using the one or more processors, a motionclass based on a motion classification model that takes as input thevertical components of inertial acceleration and rotationalacceleration, the magnitude of rotation rate, the correlation betweenthe vertical components of the inertial acceleration and rotationalacceleration and the percentage of motion outside the dominant motionplane; determining, using the one or more processors, a likelihood thatthe user is walking; in accordance with determining that the user islikely not walking, configurating, using the one or more processors, aheart rate model based on the predicted motion class; and estimating,using the configured heart rate model, a caloric expenditure of theuser.
 2. The method of claim 1, wherein determining, using the one ormore processors, a likelihood the user is walking, further comprises:obtaining, from a digital pedometer, a step count; determining an armpose of the user; and determining whether or not the user is walkingbased on the step count and arm pose.
 3. The method of claim 1, whereinthe motion classification model outputs one of three possible classes:arm only motion, with body motion and other motion.
 4. The method ofclaim 3, wherein the motion classification model has two parts: a firstpart that uses a logistic regression model with vertical acceleration,vertical component of the rotational acceleration, and correlationbetween the two acceleration to predict a likelihood of arm motion only,and a second part that detects a body component in the motion with amoderate likelihood from the logistic regression model, wherein if thepercentage of motion outside of the dominant plane is above a thresholdand the inertial vertical acceleration is within an expected range ofbody motion, the classification is with body motion.
 5. The method ofclaim 1, wherein configuring, using the one or more processors, a heartrate model based on the predicted motion class, further comprises:determining a first caloric expenditure based on a heart rate model; andobtaining a scale factor based on the predicted motion class; andscaling the first caloric expenditure by the scale factor to get asecond caloric expenditure specific to the motion class.
 6. The methodof claim 5, wherein the predicted motion class is an arm only motionclass and the heart rate model is an arm only heart rate model.
 7. Themethod of claim 5, wherein determining a first caloric expenditure basedon a heart rate model, further comprises: obtaining the user's age;obtaining the user's heart rate from a heart rate sensor embedded in orattached to the wearable device; obtaining the user's maximal oxygenuptake (VO₂Max); and determining the first caloric expenditure based onthe heart rate model with the user's age, the users heart rate and theuser's VO₂Max as inputs into the heart rate model.
 8. The method ofclaim 1, wherein the rotation rate is compensated for drift.
 9. Themethod of claim 1, wherein the dominant plane of motion is determinedusing principle component analysis (PCA) of a crown vector of thewearable device, where the dominant plane is determined by first andsecond PCA components and the percentage of motion outside of thedominant plane is a fraction of a variance in a third PCA component,which is a ratio of the variance in the third component and a totalvariance.
 10. A system comprising: motion sensors; one or moreprocessors; memory storing instructions that when executed by the one ormore processors, cause the one or more processors to perform operationscomprising: obtaining acceleration and rotation rate from motion sensorsof a wearable device worn on a limb of a user while the user is engagedin a physical activity; determining a vertical component of inertialacceleration and a vertical component of rotational acceleration fromthe acceleration and rotation rate, respectively; determining amagnitude of the rotation rate; determining a correlation between theinertial vertical acceleration component and rotational acceleration;determining a percentage of motion outside a dominant plane of motion;predicting a motion class based on a motion classification model thattakes as input the vertical components of inertial acceleration androtational acceleration, the magnitude of rotation rate, the correlationbetween the vertical components of the inertial acceleration androtational acceleration and the percentage of motion outside thedominant motion plane; determining a likelihood that the user iswalking; in accordance with determining that the user is likely notwalking, configuring a heart rate model based on the predicted motionclass; and estimating, using the configured heart rate model, a caloricexpenditure of the user.
 11. The system of claim 10, whereindetermining, using the one or more processors, a likelihood the user iswalking, further comprises: obtaining, from a digital pedometer, a stepcount; determining an arm pose of the user; and determining whether ornot the user is walking based on the step count and arm pose.
 12. Thesystem of claim 10, wherein the motion classification model outputs oneof three possible classes: arm only motion, with body motion and othermotion.
 13. The system of claim 12, wherein the motion classificationmodel has two parts: a first part that uses a logistic regression modelwith vertical acceleration, vertical component of the rotationalacceleration, and correlation between the two acceleration to predict alikelihood of arm motion only, and a second part that detects a bodycomponent in the motion with a moderate likelihood from the logisticregression model, wherein if the percentage of motion outside of thedominant plane is above a threshold and the inertial verticalacceleration is within an expected range of body motion, theclassification is with body motion.
 14. The system of claim 10, whereinconfiguring, using the one or more processors, a heart rate model basedon the predicted motion class, further comprises: determining a firstcaloric expenditure based on a heart rate model; and obtaining a scalefactor based on the predicted motion class; and scaling the firstcaloric expenditure by the scale factor to get a second caloricexpenditure specific to the motion class.
 15. The system of claim 14,wherein the predicted motion class is an arm only motion class and theheart rate model is an arm only heart rate model.
 16. The system ofclaim 14, wherein determining a first caloric expenditure based on aheart rate model, further comprises: obtaining the user's age; obtainingthe user's heart rate from a heart rate sensor embedded in or attachedto the wearable device; obtaining the user's maximal oxygen uptake(VO₂Max); and determining the first caloric expenditure based on theheart rate model with the user's age, the users heart rate and theuser's VO₂Max as inputs into the heart rate model.
 17. The system ofclaim 10, wherein the rotation rate is compensated for drift.
 18. Thesystem of claim 10, wherein the dominant plane of motion is determinedusing principle component analysis (PCA) of a crown vector of thewearable device, where the dominant plane is determined by first andsecond PCA components and the percentage of motion outside of thedominant plane is a fraction of a variance in a third PCA component,which is a ratio of the variance in the third component and a totalvariance.